Pixel interpolation method

ABSTRACT

An image processing circuit inputs pixels of an RGB Bayer array therein. A chroma value calculation circuit calculates a chroma factor (K L ) for evaluating the chroma of a surrounding area of a specified pixel. A correlation value calculation circuit calculates correlation values for gray image and color image. If the chroma factor (K L ) is larger than a threshold value (TH 1 ), a correlation judgment method for color image and a pixel interpolation method for color image are selected, if the chroma factor (K L ) is not larger than a threshold value (TH 1 ) and larger than a threshold value (TH 2 ), a correlation judgment method using a correlation value obtained by overall judgment on the correlation values for gray image and color image and a pixel interpolation method for color image are selected, and if the chroma factor (K L ) is not larger than a threshold value (TH 2 ), a correlation judgment method for gray image and a pixel interpolation method for gray image are selected.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique for pixel interpolationperformed by an image processing circuit included in a digital camera orthe like.

2. Description of the Background Art

Image pickup elements, such as CCDs, CMOSs and the like which are usedin digital cameras and the like, perform photoelectric conversion oflight received through color filters, to output pixel signals. Suchcolor filters include RGB color filters, YMCK color filters and thelike. Then, from a single-chip image pickup element, a pixel signal forone color is outputted per pixel. For example, in a case of using theRGB color filter, for one pixel, a pixel signal for one of R (Red)component, G (Green) component and B (Blue) component is outputted.

For this reason, as to the pixel signals outputted from the single-chipcolor image pickup element, an interpolation process has to be performedon pixel signals for other color components. Various algorithms are usedto perform such an interpolation process. For example, the degrees ofcorrelation in vertical and horizontal directions are calculated andpixel interpolation is performed by using the pixels in the directionwhere the degree of correlation is higher. Alternatively, weights areassigned in accordance with the respective distances between thespecified pixel and the surrounding pixels and then pixel interpolationis performed.

In Japanese Patent Application Laid Open Gazette No. 2006-186965, a grayarea and a color area are discriminated in an image and pixelinterpolations in accordance with respective characteristics of theseareas are applied to these areas. Especially, there is some contrivanceto suppress occurrence of false colors in an area positioned at aboundary between the gray area and the color area. Specifically, thearea at the boundary between the gray area and the color area is judgedto be a gray image as to the correlation direction and the pixelinterpolation for color image is applied thereto.

By using the pixel interpolation method disclosed in Japanese PatentApplication Laid Open Gazette No. 2006-186965, it is possible to reducethe false colors caused by the pixel interpolation. In other words, inthe area at the boundary between the gray area and the color area,paying attention to that respective pixel values of color components ofRGB are approximate to one another, the correlation direction is judgedby using pixels closer to the specified pixel, without distinction ofRGB. On the other hand, the pixel interpolation is performed withdistinction of RGB, to achieve an interpolation result with highprecision.

Thus, the reason why such a contrivance for the pixel interpolation hasto be performed especially in the area at the boundary between a grayimage and a color image is that there is a possibility that the falsecolors may occur in high-frequency components such as fine lines in ahorizontal or vertical direction in this area or the like.

SUMMARY OF THE INVENTION

The present invention is intended for a pixel interpolation method.According to an aspect of the present invention, the pixel interpolationmethod comprises the steps of (a) inputting a pixel signal of apredetermined color space, (b) calculating a chroma evaluation value ofan area consisting of a specified pixel and its surrounding pixels, (c)selecting a correlation judgment method and a pixel interpolation methodon the specified pixel on the basis of the chroma evaluation value, and(d) performing a pixel interpolation process on the specified pixel in acorrelation direction determined by the selected correlation judgmentmethod, by using the selected pixel interpolation method, and in thepresent method, by using predetermined two threshold values TH1 and TH2(TH1≧TH2), if the chroma evaluation value is not larger than thethreshold value TH2, a correlation judgment method for gray image and apixel interpolation method for gray image are selected, if the chromaevaluation value is larger than the threshold value TH2 and not largerthan the threshold value TH1, a correlation judgment method using acorrelation value selected out of correlation values for gray image andcolor image and a pixel interpolation method for color image areselected, and if the chroma evaluation value is larger than thethreshold value TH1, a correlation judgment method for color image and apixel interpolation method for color image are selected in the step (c).

By the present invention, it is possible to prevent wrong judgment onthe correlation direction and wrong interpolation and reduce falsecolors occurring in the pixel interpolation process.

According to another aspect of the present invention, the step (c)includes the step of (c-1) selecting a correlation judgment method usinga correlation value out of correlation values for gray image and colorimage, which is judged that the correlation on the specified pixel isreflected higher on, if the chroma evaluation value is larger than thethreshold value TH2 and not larger than the threshold value TH1.

It is thereby possible to improve the accuracy in judgment on thecorrelation direction.

Therefore, it is an object of the present invention to provide atechnique to further effectively reduce false colors caused by pixelinterpolation in an area at the boundary between a gray image and acolor image.

These and other objects, features, aspects and advantages of the presentinvention will become more apparent from the following detaileddescription of the present invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a digital camera in accordance withthe preferred embodiments;

FIGS. 2A to 2E are views each showing an array pattern of pixels in aRGB Bayer array;

FIG. 3 is a graph showing a relation between a chroma value and a chromafactor;

FIG. 4 is a view showing four correlation directions;

FIGS. 5A and 5B are views each showing a correlation value calculationmethod in a vertical direction in a color area where a specified pixelis a G pixel;

FIGS. 6A and 6B are views each showing a correlation value calculationmethod in a horizontal direction in the color area where the specifiedpixel is a G pixel;

FIGS. 7A to 7C are views each showing a correlation value calculationmethod in a diagonal A direction in the color area where the specifiedpixel is a G pixel;

FIGS. 8A to 8C are views each showing a correlation value calculationmethod in a diagonal B direction in the color area where the specifiedpixel is a G pixel;

FIGS. 9A and 9B are views each showing a correlation value calculationmethod in the vertical direction in a color area where the specifiedpixel is an R pixel or a B pixel;

FIGS. 10A and 10B are views each showing a correlation value calculationmethod in the horizontal direction in the color area where the specifiedpixel is an R pixel or a B pixel;

FIGS. 11A to 11C are views each showing a correlation value calculationmethod in the diagonal A direction in the color area where the specifiedpixel is an R pixel or a B pixel;

FIGS. 12A to 12C are views each showing a correlation value calculationmethod in the diagonal B direction in the color area where the specifiedpixel is an R pixel or a B pixel;

FIG. 13 is a view showing a correlation value calculation method in thevertical direction in a gray area;

FIG. 14 is a view showing a correlation value calculation method in thehorizontal direction in the gray area;

FIG. 15 is a view showing a correlation value calculation method in thediagonal A direction in the gray area;

FIG. 16 is a view showing a correlation value calculation method in thediagonal B direction in the gray area;

FIG. 17 is a view showing a criterion of judgment on a correlationjudgment method and a pixel interpolation method;

FIG. 18 is a graph showing a correspondence where the correlation isevaluated highly in the vertical and horizontal directions;

FIG. 19 is a graph showing a correspondence where the correlation isevaluated highly in the diagonal A and diagonal B directions;

FIG. 20 is a graph showing a correspondence where the correlation isevaluated low in the vertical and horizontal directions;

FIG. 21 is a graph showing a correspondence where the correlation isevaluated low in the diagonal A and diagonal B directions; and

FIG. 22 is a view showing a criterion of judgment on a correlationjudgment method and a pixel interpolation method in a second preferredembodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, the preferred embodiments of the present invention will bediscussed with reference to figures.

The First Preferred Embodiment

<1. Schematic Overall Structure of Digital Camera>

FIG. 1 is a block diagram showing a digital camera 10 in accordance withthe present invention. The digital camera 10 comprises an image pickupelement 1, a signal processing circuit 2, an image processing circuit 3and a memory 5. The image pickup element 1 is a single-chip CCDcomprising a color filter array of RGB Bayer array and outputs a pixelsignal of any one of color components of R (Red), G (Green) and B (Blue)from one pixel. Specifically, for example, if G signals and R signalsare alternately outputted, such as G→R→G→R . . . , in horizontal linesin odd rows, B signals and G signals are alternately outputted, such asB→G→B→G . . . , in horizontal lines in even rows. Further, as the imagepickup element 1, a CMOS sensor may be used.

The pixel signal outputted from the image pickup element 1 is inputtedto the signal processing circuit 2. In the signal processing circuit 2,a signal processing such as white balancing, black level correction orthe like is performed on the pixel signal. The pixel signal outputtedfrom the signal processing circuit 2 is inputted to the image processingcircuit 3. The image processing circuit 3 comprises a chroma valuecalculation circuit 31, a correlation value calculation circuit 32, aselection circuit 33, first and second correlation judgment circuits 341and 342, first and second interpolation circuits 351 and 352 and firstand second color space conversion circuits 361 and 362.

The chroma value calculation circuit 31 uses the pixel signals of thespecified pixel and its surrounding pixels to calculate a chroma valueof this area. This chroma value serves as an indicator for judgment onwhether the area is a gray image or a color image.

The correlation value calculation circuit 32 uses the pixel signals ofthe specified pixel and its surrounding pixels to calculate acorrelation value of this area,

The selection circuit 33 selects whether an operation for gray image oran operation for color image is to be performed in a correlationjudgment process and a pixel interpolation process, on the basis of thechroma value calculated by the chroma value calculation circuit 31.

The first correlation judgment circuit 341 and the second correlationjudgment circuit 342 each use the correlation value which is calculatedby the correlation value calculation circuit 32 and selected by theselection circuit 33, to judge a correlation direction.

The first interpolation circuit 351 performs the pixel interpolationprocess on the specified pixel on the basis of the judgment result ofthe first correlation judgment circuit 341, and the second interpolationcircuit 352 performs the pixel interpolation process on the specifiedpixel on the basis of the judgment result of the second correlationjudgment circuit 342.

The first color space conversion circuit 361 performs color spaceconversion of the pixel signal of RGB interpolated by the firstinterpolation circuit 351, to generate a Y signal. The second colorspace conversion circuit 362 performs color space conversion of thepixel signal of RGB interpolated by the second interpolation circuit352, to generate a Cb signal and a Cr signal.

Further, the chroma value calculation circuit 31, the correlation valuecalculation circuit 32 and the first and second interpolation circuits351 and 352 each comprise a group of registers for accumulating pixelsignals in a matrix area of M×N in order to perform computation usingthe pixel signals of the specified pixel and its surrounding pixels.Furthermore, these circuits 31, 32, 351 and 352 may share the registers.

After the pixel interpolation process is performed in the first andsecond interpolation circuits 351 and 352, each pixel becomes a signalhaving all the color components of RGB and is converted into a YCbCrsignal by the first and second color space conversion circuits 361 and362. Then, the pixel signal is stored into the memory 5.

<2. Representation of Pixels in Bayer Array>

Next, discussion will be made on representation of pixels in Bayer arrayused in the following description and figures. The pixels in a matrixarea of 5×5 are represented in FIG. 2A. In FIG. 2A, reference sign ‘P’represents a pixel without consideration of color components of RGB. Incontrast to this, in FIGS. 2B to 2E, pixels are represented withdistinction of color components. Reference signs R, G and B represent ared pixel, a green pixel and a blue pixel, respectively. Further, inFIGS. 2 and 5 to 16, G pixels are represented by solid circles and R andB pixels are represented by broken-line circles.

Among two numbers following the reference signs P, R, G and B, the firstnumber represents the row number of a pixel in the matrix area and thesecond number represents the column number of the pixel in the matrixarea. FIGS. 2A to 2E each represent a pixel array of the matrix areaconsisting of 25 pixels, P00 to P44, including a specified pixel P22.The same representation is applied to other figures. Further, indescription of the preferred embodiments and equations, the referencesigns P, R, G and B sometimes represent pixel values. The reference signP11, for example, represents a pixel at the first row and the firstcolumn and also represents the pixel value of the pixel at the first rowand the first column.

FIGS. 2B and 2E represent pixel arrays in a case where the specifiedpixel P22 is a G signal. FIG. 2C represents a pixel array in a casewhere the specified pixel P22 is an R signal. FIG. 2D represents a pixelarray in a case where the specified pixel P22 is a B signal. Asdiscussed above, in the chroma value calculation circuit 31, thecorrelation value calculation circuit 32 and the first and secondinterpolation circuits 351 and 352, the pixel signals in the matrix areaare accumulated in the group of registers in order to performcomputation using the pixel signals of the specified pixel and itssurrounding pixels. In a case where pixels in a matrix area of 5×5 areto be processed, there are four patterns of pixel signals stored in thegroup of registers as shown in FIGS. 2B to 2E. Further, in a case wherepixels in a matrix area of 3×3 are to be processed, nine pixels P11,P12, P13, P21, P22, P23, P31, P32 and P33, centering the specified pixelP22, are used and there are also four patterns of pixel signals as shownin FIGS. 2B to 2E.

<3. Chroma Value Calculation Process>

Next, detailed discussion will be made on a chroma value calculationprocess performed by the chroma value calculation circuit 31. The chromavalue calculation circuit 31 analyzes a color difference component in amatrix area (consisting of a specified pixel and its surrounding pixels)including a specified pixel and calculates a chroma evaluation value ofthis area. This chroma evaluation value is used later in a selectionstep to judge whether an area to be processed is an image with highchroma (hereinafter, referred to as color image) or an image with lowchroma (hereinafter, referred to as gray image).

The chroma evaluation value is calculated on the basis of a colordifference component between the level of G pixel and the level of Rpixel and a color difference component between the level of G pixel andthe level of B pixel. In the first preferred embodiment, two colordifference component evaluation values are calculated in order todetermine the chroma evaluation value. In other words, the chroma valuecalculation circuit 31 calculates a “first color difference componentevaluation value” and a “second color difference component evaluationvalue”. The “first color difference component evaluation value” refersto an evaluation value obtained from a color difference component valueon the basis of respective average pixel values for color components ofthe pixels existing in this area without consideration of the respectivepositions of the pixels in the matrix area. The “second color differencecomponent evaluation value” refers to an evaluation value obtained byaccumulating the color difference component values in a specificdirection, in consideration of the respective positions of the pixels inthe matrix area.

Thus, the reason why two kinds of color difference component evaluationvalues are calculated is as follows. In a gray image in which fine linesare present in a horizontal or vertical direction, such as retoma chart,if the above “first color difference component evaluation value” isadopted as the chroma value, there is a possibility that the gray imagemay be wrongly judged as a color image. The reason of this phenomenon isthat even though there is a strong correlation in the horizontal orvertical direction, the color difference component is calculated byusing the average pixel values in this area without consideration ofthis correlation. Then, in the first preferred embodiment, as discussedbelow, two kinds of color difference component evaluation values arecalculated and the lower one in level of color difference component isadopted as the chroma value.

(3-1) The First Color Difference Component Evaluation Value

First, discussion will be made on a method of calculating the firstcolor difference component evaluation value. The first color differencecomponent evaluation value is suitable for evaluation of colordifference components of flat parts (low-frequency areas) such as bluesky, wallpaper without any pattern, or the like. Herein, the flat partrefers to an area having no strong correlation in a specific direction.In order to calculate the first color difference component evaluationvalue, first, respective average values R_(ave), G_(ave) and B_(ave) ofthe pixel values for R, G and B included in a matrix area of 3×3centering a specified pixel are calculated. The average values R_(ave),G_(ave) and B_(ave) are generally expressed as Eq. 1. In Eq. 1, N_(R),N_(G) and N_(B) represent the numbers of pixels of R, G and B,respectively, existing in this matrix area and the terms of Σ representrespective cumulative pixel values for the three color components.

$\begin{matrix}\{ \begin{matrix}{R_{ave} = {\frac{1}{N_{R}}{\sum\limits_{i}^{N_{R}}R_{i}}}} \\{G_{ave} = {\frac{1}{N_{G}}{\sum\limits_{i}^{N_{G}}G_{i}}}} \\{B_{ave} = {\frac{1}{N_{B}}{\sum\limits_{i}^{N_{B}}B_{i}}}}\end{matrix}  & ( {{Eq}.\mspace{14mu} 1} )\end{matrix}$

As shown in FIGS. 2B to 2E, however, since there are four patterns ofpixel arrays, the method of calculating the average value is differentfrom pattern to pattern. First, in a case where the center pixel is a Gpixel and the pattern corresponds to the pixel array of FIG. 2B, theaverage values R_(ave), G_(ave) and B_(ave) are calculated from Eq. 2.

$\begin{matrix}{{G_{ave} = \frac{{G\; 11} + {G\; 13} + {G\; 22} + {G\; 31} + {G\; 33}}{5}}{R_{ave} = \frac{{R\; 21} + {R\; 23}}{2}}{B_{ave} = \frac{{B\; 12} + {B\; 32}}{2}}} & ( {{Eq}.\mspace{14mu} 2} )\end{matrix}$

In a case where the center pixel is an R pixel and the patterncorresponds to the pixel array of FIG. 2C, the average values R_(ave),G_(ave) and B_(ave) are calculated from Eq. 3.

$\begin{matrix}{{G_{ave} = \frac{{G\; 12} + {G\; 21} + {G\; 23} + {G\; 32}}{4}}{R_{ave} = {R\; 22}}{B_{ave} = \frac{{B\; 11} + {B\; 13} + {B\; 31} + {B\; 33}}{4}}} & ( {{Eq}.\mspace{14mu} 3} )\end{matrix}$

In a case where the center pixel is a B pixel and the patterncorresponds to the pixel array of FIG. 2D, the average values R_(ave),G_(ave) and B_(ave) are calculated from Eq. 4.

$\begin{matrix}{{G_{ave} = \frac{{G\; 12} + {G\; 21} + {G\; 23} + {G\; 32}}{4}}{R_{ave} = \frac{{R\; 11} + {R\; 13} + {R\; 31} + {R\; 33}}{4}}{B_{ave} = {B\; 22}}} & ( {{Eq}.\mspace{14mu} 4} )\end{matrix}$

In a case where the center pixel is a G pixel and the patterncorresponds to the pixel array of FIG. 2E, the average values R_(ave),G_(ave) and B_(ave) are calculated from Eq. 5.

$\begin{matrix}{{G_{ave} = \frac{{G\; 11} + {G\; 13} + {G\; 22} + {G\; 31} + {G\; 33}}{5}}{R_{ave} = \frac{{R\; 12} + {R\; 32}}{2}}{B_{ave} = \frac{{B\; 21} + {B\; 23}}{2}}} & ( {{Eq}.\mspace{14mu} 5} )\end{matrix}$

The chroma value calculation circuit 31 performs computation inaccordance with any one of Eqs. 2 to 5 depending on which one of thepatterns shown in FIGS. 2B to 2E the pixel array of the matrix area is,to calculate the average values R_(ave), G_(ave) and B_(ave). The chromavalue calculation circuit 31 further performs computation expressed byEq. 6 by using the calculated average values R_(ave), G_(ave) andB_(ave), to calculate the first color difference component evaluationvalue L_(global). In other words, the color difference componentevaluation value L_(global) is an evaluation value of color differencecomponent, which is calculated by using the color difference componentvalue on the basis of the respective average pixel values for the colorcomponents existing in the matrix area.

$\begin{matrix}{L_{global} = \frac{{{G_{ave} - R_{ave}}} + {{G_{ave} - B_{ave}}}}{2}} & ( {{Eq}.\mspace{14mu} 6} )\end{matrix}$

(3-2) The Second Color Difference Component Evaluation Value

Next, discussion will be made on a method of calculating the secondcolor difference component evaluation value. The second color differencecomponent evaluation value is suitable for evaluation of colordifference components in an area where there is a strong correlation ina matrix area and the chroma value may largely vary depending on themethod of calculating the color difference component value. As discussedabove, for example, if the first color difference component evaluationvalue obtained in (3-1) is adopted as the chroma evaluation value in agray image including a high-frequency component, such as a retoma chartor the like, there is a possibility that the gray image may be wronglyjudged as a color image. Then, in order to appropriately obtain thecolor difference component evaluation value for such an image having astrong correlation in a specific direction, the following operation isperformed.

The chroma value calculation circuit 31 performs computation expressedby Eqs. 7 and 8 by using pixel signals in a matrix area of 3×3.Specifically, in Eq. 7, the color difference component values areaccumulated in the vertical direction, to calculate the color differencecomponent evaluation value L_(vertical) in the vertical direction.Further, in Eq. 8, the color difference component values are accumulatedin the horizontal direction, to calculate the color difference componentevaluation value L_(horizontal) in the horizontal direction. In otherwords, in the computations of Eqs. 7 and 8, the color differencecomponent values of the G pixels and the R pixels and the colordifference component values of the G pixels and the B pixels areaccumulated in the vertical direction and the horizontal direction,respectively. Both the color difference component evaluation valuesL_(vertical) and L_(horizontal) are the above-discussed second colordifference component evaluation values.

$\begin{matrix}{L_{vertical} = \frac{\begin{matrix}{{{{P\; 11} - {P\; 21}}} + {{{P\; 21} - {P\; 31}}} + {2{{{P\; 12} - {P\; 22}}}} +} \\{{2{{{P\; 22} - {P\; 32}}}} + {{{P\; 13} - {P\; 23}}} + {{{P\; 23} - {P\; 33}}}}\end{matrix}}{8}} & ( {{Eq}.\mspace{14mu} 7} ) \\{L_{horizontal} = \frac{\begin{matrix}{{{{P\; 11} - {P\; 12}}} + {{{P\; 12} - {P\; 13}}} + {2{{{P\; 21} - {P\; 22}}}} +} \\{{2{{{P\; 22} - {P\; 23}}}} + {{{P\; 31} - {P\; 32}}} + {{{P\; 32} - {P\; 33}}}}\end{matrix}}{8}} & ( {{Eq}.\mspace{14mu} 8} )\end{matrix}$

Further, in Eqs. 7 and 8, there are terms multiplied by a coefficient of“2”. This is for coincidence between the color difference componentcumulative number of the G and R pixels and that of the G and B pixels.Though the coefficient of “2” is used as a multiplier for coincidencebetween the cumulative numbers of different color difference componentsin the first preferred embodiment, however, the value of thiscoefficient may be set as appropriate.

Further, though the color difference component evaluation values in thehorizontal and vertical directions are calculated in the first preferredembodiment, color difference component evaluation values in diagonaldirections may be additionally calculated as objects of evaluation. Forexample, a color difference component evaluation value Ld_(A) of adiagonal A direction which has the inclination of 45 degrees clockwiselyfrom the horizontal direction with respect to the specified pixel P22and a color difference component evaluation value Ld_(B) of a diagonal Bdirection orthogonal to the diagonal A direction can be obtained fromEqs. 9 and 10. Eq. 9 is an equation, however, which is used in the casewhere the specified pixel P22 is a G pixel and Eq. 10 is an equationwhich is used in the case where the specified pixel P22 is an R pixel ora B pixel.

$\begin{matrix}{{{Ld}_{A} = {\begin{pmatrix}{{{\frac{{P\; 13} + {P\; 22}}{2} - {P\; 12}}} + {{\frac{{P\; 13} + {P\; 22}}{2} - {P\; 23}}} +} \\{{{\frac{{P\; 31} + {P\; 22}}{2} - {P\; 21}}} + {{\frac{{P\; 31} + {P\; 22}}{2} - {P\; 32}}}}\end{pmatrix} \times \frac{1}{4}}}{{Ld}_{B} = {\begin{pmatrix}{{{\frac{{P\; 11} + {P\; 22}}{2} - {P\; 12}}} + {{\frac{{P\; 11} + {P\; 22}}{2} - {P\; 21}}} +} \\{{{\frac{{P\; 33} + {P\; 22}}{2} - {P\; 23}}} + {{\frac{{P\; 33} + {P\; 22}}{2} - {P\; 32}}}}\end{pmatrix} \times \frac{1}{4}}}} & ( {{Eq}.\mspace{14mu} 9} ) \\{{{Ld}_{A} = {\begin{pmatrix}{{{\frac{{P\; 12} + {P\; 21}}{2} - {P\; 11}}} + {{\frac{{P\; 12} + {P\; 21}}{2} - {P\; 22}}} +} \\{{{\frac{{P\; 23} + {P\; 32}}{2} - {P\; 21}}} + {{\frac{{P\; 23} + {P\; 32}}{2} - {P\; 33}}}}\end{pmatrix} \times \frac{1}{4}}}{{Ld}_{B} = {\begin{pmatrix}{{{\frac{{P\; 12} + {P\; 23}}{2} - {P\; 13}}} + {{\frac{{P\; 12} + {P\; 23}}{2} - {P\; 22}}} +} \\{{{\frac{{P\; 21} + {P\; 32}}{2} - {P\; 22}}} + {{\frac{{P\; 21} + {P\; 32}}{2} - {P\; 31}}}}\end{pmatrix} \times \frac{1}{4}}}} & ( {{Eq}.\mspace{14mu} 10} )\end{matrix}$

(3-3) Calculation of Chroma Factor

After calculating the three color difference component evaluation valuesL_(global), L_(vertical) and L_(horizontal) by using the calculationmethods as shown above in (3-1) and (3-2), the chroma value calculationcircuit 31 further performs computation expressed by Eq. 11, tocalculate a minimum value of the color difference component evaluationvalues L_(global), L_(vertical) and L_(horizontal) (in other words, thesmallest one in level of the color difference component). This minimumvalue is adopted as a chroma evaluation value L of the matrix area to beprocessed. In other words, the chroma evaluation value L is a chromavalue determined in accordance with the specified pixel. In Eq. 11, min(x, y, z) represents the minimum value of x, y and z. Further, asdiscussed above, the color difference component evaluation values in thediagonal directions may be calculated as the second color differencecomponent evaluation values as well as L_(vertical) and L_(horizontal),and in such a case, the minimum value has to be selected among theevaluation values including the color difference component evaluationvalues in the diagonal directions.L=min(L _(global) ,L _(horizontal) ,L _(vertical))  (Eq. 11)

After performing the above computations to obtain the chroma evaluationvalue L with respect to the specified pixel, the chroma valuecalculation circuit 31 next normalizes the chroma evaluation value L tocalculate a chroma factor K_(L). Specifically, the chroma valuecalculation circuit 31 uses two threshold values T₁ and T₂ to performnormalization as expressed by Eq. 12. FIG. 3 is a graph showing arelation between the chroma evaluation value L and the chroma factorK_(L). As shown in FIG. 3, the chroma factor K_(L) for judgment onwhether a gray image or a color image gently changes between thethreshold values T₁ and T₂ which are set near the area where there is achange from a gray image to a color image, to ease a sharp change ofimage judgment.

$\begin{matrix}\begin{matrix}{{{When}\mspace{14mu} L} \leqq T_{1}} & {K_{L} = 0} \\{{{When}\mspace{14mu} T_{1}} < L < T_{2}} & {K_{L} = \frac{L - T_{1}}{T_{2} - T_{1}}} \\{{{When}\mspace{14mu} T_{2}} \leqq L} & {K_{L} = 1}\end{matrix} & ( {{Eq}.\mspace{14mu} 12} )\end{matrix}$

Further, since the two threshold values T₁ and T₂ are set near theboundary between the gray image and the color image, these values haveonly to be determined appropriately on the basis of experimental resultsor experience but preferably should be variable parameters depending oncharacteristics of an input image. The characteristics of an input imageare determined on the basis of photographing conditions such as theexposure time, the aperture and the like. Further, besides thecharacteristics of an input image, the characteristics of a CCD, opticalcharacteristics of a lens and the like may be taken into account. Thechroma factor K_(L) which is calculated thus is used later in theselection step.

<4. Correlation Value Calculation Process>

The correlation value calculation circuit 32 uses the pixel signals ofthe specified pixel and its surrounding pixels to calculate correlationvalues in four directions in the matrix area. Herein, as shown in FIG.4, the correlation values are calculated in the horizontal direction,the vertical direction, the diagonal A direction having the inclinationof 45 degrees clockwisely with respect to the horizontal direction andthe diagonal B direction orthogonal to the diagonal A direction.Specifically, a pixel differential value which is a difference betweenthe values of the pixels existing in each of these four directions iscalculated and the pixel differential values in each direction areaccumulated, to obtain the correlation value.

Further, in the first preferred embodiment, the correlation valuecalculation circuit 32 calculates both a correlation value for colorimage having high chroma and a correlation value for gray image havinglow chroma. Then, finally in the selection process of the later step,either one of the correlation values for color image and gray image isselected and a correlation direction is determined. Alternatively, acorrelation value selected by overall judgment on both the correlationvalues for color image and the gray image is used to determine thecorrelation direction.

(4-1) Correlation Value for Color Image Having High Chroma

(4-1-1) Case where Center Pixel is G

First, discussion will be made on a correlation value calculation methodfor color image in a case where the specified pixel is a G pixel. Inother words, this is a correlation value calculation method in a casewhere the matrix area has such a pixel array as shown in FIG. 2B or 2E.A correlation value in the vertical direction is calculated from Eq. 13.FIGS. 5A and 5B are views each showing a correlation value calculationmethod in the vertical direction, and FIG. 5A shows a correlation valuecalculation method on G pixels and FIG. 5B shows a correlation valuecalculation method on R pixels and B pixels. Thus, the correlation valuefor color image takes pixel differential values of all the colorcomponents into account. In FIGS. 5 to 16, two pixels connected with anarrow are objects for calculation of the pixel differential value.

$\begin{matrix}{{Cv\_ c} = {( {{{{P\; 02} - {P\; 22}}} + {{{P\; 22} - {P\; 42}}} + {{{P\; 11} - {P\; 31}}} + {{{P\; 13} - {P\; 33}}} + {{{P\; 00} - {P\; 20}}} + {{{P\; 20} - {P\; 40}}} + {{{P\; 04} - {P\; 24}}} + {{{P\; 24} - {P\; 44}}} + {{{P\; 12} - {P\; 32}}} + {{{P\; 10} - {P\; 30}}} + {{{P\; 14} - {P\; 34}}} + {{{P\; 01} - {P\; 21}}} + {{{P\; 21} - {P\; 41}}} + {{{P\; 03} - {P\; 23}}} + {{{P\; 23} - {P\; 43}}}} ) \times \frac{1}{15}}} & ( {{Eq}.\mspace{14mu} 13} )\end{matrix}$

A correlation value in the horizontal direction is calculated from Eq.14. FIGS. 6A and 6B are views each showing a correlation valuecalculation method in the horizontal direction, and FIG. 6A shows acorrelation value calculation method on G pixels and FIG. 6B shows acorrelation value calculation method on R pixels and B pixels.

$\begin{matrix}{{Ch\_ c} = {( {{{{P\; 20} - {P\; 22}}} + {{{P\; 22} - {P\; 24}}} + {{{P\; 11} - {P\; 13}}} + {{{P\; 31} - {P\; 33}}} + {{{P\; 00} - {P\; 02}}} + {{{P\; 02} - {P\; 04}}} + {{{P\; 40} - {P\; 42}}} + {{{P\; 42} - {P\; 44}}} + {{{P\; 21} - {P\; 23}}} + {{{P\; 01} - {P\; 03}}} + {{{P\; 41} - {P\; 43}}} + {{{P\; 10} - {P\; 12}}} + {{{P\; 12} - {P\; 14}}} + {{{P\; 30} - {P\; 32}}} + {{{P\; 32} - {P\; 34}}}} ) \times \frac{1}{15}}} & ( {{Eq}.\mspace{14mu} 14} )\end{matrix}$

A correlation value in the diagonal A direction is calculated from Eq.15. FIGS. 7A to 7C are views each showing a correlation valuecalculation method in the diagonal A direction, and FIG. 7A shows acorrelation value calculation method on G pixels and FIGS. 7B and 7Cshow correlation value calculation methods on R pixels and B pixels,respectively.

$\begin{matrix}{{{Cd}_{A}{\_ c}} = {( {{{{{P\; 11} - {P\; 22}}} \times 2} + {{{{P\; 22} - {P\; 33}}} \times 2} + {{{{P\; 02} - {P\; 13}}} \times 2} + {{{{P\; 13} - {P\; 24}}} \times 2} + {{{{P\; 20} - {P\; 31}}} \times 2} + {{{{P\; 31} - {P\; 42}}} \times 2} + {{{{P\; 00} - {P\; 11}}} \times 2} + {{{{P\; 33} - {P\; 44}}} \times 2} + {{{P\; 01} - {P\; 23}}} + {{{P\; 21} - {P\; 43}}} + {{{P\; 12} - {P\; 34}}} + {{{P\; 10} - {P\; 32}}}} ) \times \frac{1}{12}}} & ( {{Eq}.\mspace{14mu} 15} )\end{matrix}$

As shown in FIGS. 7A to 7C, between the case of calculation on G pixelsand the case of calculation on R pixels or B pixels, there is adifference in the distance between pixels on which the differentialvalue is calculated. Then, in Eq. 15, the differential value on two Gpixels with short distance is multiplied by 2. The reason is that thedistance between two G pixels which are objects for calculation is halfthe distance between two R pixels or two B pixels which are objects forcalculation and the pixel differential value responds to the amount ofvariation on G pixels that is twice as large as that on R pixels or Bpixels. The multiplier of 2, however, is an example and may be selectedas appropriate.

A correlation value in the diagonal B direction is calculated from Eq.16. FIGS. 8A to 8C are views each showing a correlation valuecalculation method in the diagonal B direction, and FIG. 8A shows acorrelation value calculation method on G pixels and FIGS. 8B and 8Cshow correlation value calculation methods on R pixels and B pixels,respectively.

$\begin{matrix}{{{Cd}_{B}{\_ c}} = {( {{{{{P\; 13} - {P\; 22}}} \times 2} + {{{{P\; 22} - {P\; 31}}} \times 2} + {{{{P\; 02} - {P\; 11}}} \times 2} + {{{{P\; 11} - {P\; 20}}} \times 2} + {{{{P\; 24} - {P\; 33}}} \times 2} + {{{{P\; 33} - {P\; 42}}} \times 2} + {{{{P\; 04} - {P\; 13}}} \times 2} + {{{{P\; 31} - {P\; 40}}} \times 2} + {{{P\; 03} - {P\; 21}}} + {{{P\; 23} - {P\; 41}}} + {{{P\; 12} - {P\; 30}}} + {{{P\; 14} - {P\; 32}}}} ) \times \frac{1}{12}}} & ( {{Eq}.\mspace{14mu} 16} )\end{matrix}$

Also in Eq. 16, like in Eq. 15, the differential value on two G pixelsis multiplied by 2. Further, though the distance between pixels in FIGS.7B, 7C, 8B and 8C is different from that in FIGS. 5A, 5B, 6A and 6B,herein the distance is regarded as equal, as the distance between twopixels with one pixel inserted therebetween. The differential value maybe multiplied by a coefficient, however, taking the distance between thepixels into account.

(4-1-2) Case where Center Pixel is B or R

Next, discussion will be made on a correlation value calculation methodfor color image in a case where the specified pixel is a B or R pixel.In other words, this is a correlation value calculation method in a casewhere the matrix area has such a pixel array as shown in FIG. 2C or 2D.A correlation value in the vertical direction is calculated from Eq. 17.FIGS. 9A and 9B are views each showing a correlation value calculationmethod in the vertical direction, and FIG. 9A shows a correlation valuecalculation method on G pixels and FIG. 9B shows a correlation valuecalculation method on R pixels and B pixels.

$\begin{matrix}{{Cv\_ c} = {( {{{{P\; 12} - {P\; 32}}} + {{{P\; 01} - {P\; 21}}} + {{{P\; 21} - {P\; 41}}} + {{{P\; 03} - {P\; 23}}} + {{{P\; 23} - {P\; 43}}} + {{{P\; 10} - {P\; 30}}} + {{{P\; 14} - {P\; 34}}} + {{{P\; 11} - {P\; 31}}} + {{{P\; 13} - {P\; 33}}} + {{{P\; 02} - {P\; 22}}} + {{{P\; 22} - {P\; 42}}} + {{{P\; 00} - {P\; 20}}} + {{{P\; 20} - {P\; 40}}} + {{{P\; 04} - {P\; 24}}} + {{{P\; 24} - {P\; 44}}}} ) \times \frac{1}{15}}} & ( {{Eq}.\mspace{14mu} 17} )\end{matrix}$

A correlation value in the horizontal direction is calculated from Eq.18. FIGS. 10A and 10B are views each showing a correlation valuecalculation method in the horizontal direction, and FIG. 10A shows acorrelation value calculation method on G pixels and FIG. 10B shows acorrelation value calculation method on R pixels and B pixels.

$\begin{matrix}{{Ch\_ c} = {( {{{{P\; 21} - {P\; 23}}} + {{{P\; 10} - {P\; 12}}} + {{{P\; 12} - {P\; 14}}} + {{{P\; 30} - {P\; 32}}} + {{{P\; 32} - {P\; 34}}} + {{{P\; 01} - {P\; 03}}} + {{{P\; 41} - {P\; 43}}} + {{{P\; 11} - {P\; 13}}} + {{{P\; 31} - {P\; 33}}} + {{{P\; 20} - {P\; 22}}} + {{{P\; 22} - {P\; 24}}} + {{{P\; 00} - {P\; 02}}} + {{{P\; 02} - {P\; 04}}} + {{{P\; 40} - {P\; 42}}} + {{{P\; 42} - {P\; 44}}}} ) \times \frac{1}{15}}} & ( {{Eq}.\mspace{14mu} 18} )\end{matrix}$

A correlation value in the diagonal A direction is calculated from Eq.19. FIGS. 11A to 11C are views each showing a correlation valuecalculation method in the diagonal A direction, and FIG. 11A shows acorrelation value calculation method on G pixels and FIGS. 11B and 11Cshow correlation value calculation methods on R pixels and B pixels,respectively.

$\begin{matrix}{{{Cd}_{A}{\_ c}} = {( {{{{{P\; 12} - {P\; 23}}} \times 2} + {{{{P\; 21} - {P\; 32}}} \times 2} + {{{{P\; 01} - {P\; 12}}} \times 2} + {{{{P\; 23} - {P\; 34}}} \times 2} + {{{{P\; 10} - {P\; 21}}} \times 2} + {{{{P\; 32} - {P\; 43}}} \times 2} + {{{{P\; 03} - {P\; 14}}} \times 2} + {{{{P\; 30} - {P\; 41}}} \times 2} + {{{P\; 11} - {P\; 33}}} + {{{P\; 00} - {P\; 22}}} + {{{P\; 22} - {P\; 44}}} + {{{P\; 02} - {P\; 24}}} + {{{P\; 20} - {P\; 42}}}} ) \times \frac{1}{13}}} & ( {{Eq}.\mspace{14mu} 19} )\end{matrix}$

A correlation value in the diagonal B direction is calculated from Eq.20. FIGS. 12A to 12C are views each showing a correlation valuecalculation method in the diagonal B direction, and FIG. 12A shows acorrelation value calculation method on G pixels and FIGS. 12B and 12Cshow correlation value calculation methods on R pixels and B pixels,respectively.

$\begin{matrix}{{{Cd}_{B}{\_ c}} = {( {{{{{P\; 12} - {P\; 21}}} \times 2} + {{{{P\; 23} - {P\; 32}}} \times 2} + {{{{P\; 03} - {P\; 12}}} \times 2} + {{{{P\; 21} - {P\; 30}}} \times 2} + {{{{P\; 14} - {P\; 23}}} \times 2} + {{{{P\; 32} - {P\; 41}}} \times 2} + {{{{P\; 01} - {P\; 10}}} \times 2} + {{{{P\; 34} - {P\; 43}}} \times 2} + {{{P\; 13} - {P\; 31}}} + {{{P\; 04} - {P\; 22}}} + {{{P\; 22} - {P\; 40}}} + {{{P\; 02} - {P\; 20}}} + {{{P\; 24} - {P\; 42}}}} ) \times \frac{1}{13}}} & ( {{Eq}.\mspace{14mu} 20} )\end{matrix}$

Also in Eqs. 19 and 20, as discussed in Eq. 15, the differential valueon two G pixels is multiplied by 2. Further, though the distance betweenpixels in FIGS. 11B, 11C, 12B and 12C is different from that in FIGS. 9Band 10B, herein the distance is regarded as equal, as the distancebetween two pixels with one pixel inserted therebetween. Thedifferential value may be multiplied by a coefficient, however, takingthe distance between the pixels into account.

(4-2) Correlation Value for Gray Image Having Low Chroma

For gray image having low chroma, correlation values are calculatedwithout distinction of the type of the specified pixel, i.e., R, G or B.In other words, the correlation values are calculated by the followingcommon computation, regardless of whatever the pixel array of the matrixarea is among these shown in FIGS. 2B to 2E. A correlation value in thevertical direction is calculated from Eq. 21. FIG. 13 is a view showinga correlation value calculation method in the vertical direction.

$\begin{matrix}{{Cv\_ m} = {( {{{{P\; 02} - {P\; 12}}} + {{{P\; 12} - {P\; 22}}} + {{{P\; 22} - {P\; 32}}} + {{{P\; 32} - {P\; 42}}} + {{{P\; 01} - {P\; 11}}} + {{{P\; 11} - {P\; 21}}} + {{{P\; 21} - {P\; 31}}} + {{{P\; 31} - {P\; 41}}} + {{{P\; 03} - {P\; 13}}} + {{{P\; 13} - {P\; 23}}} + {{{P\; 23} - {P\; 33}}} + {{{P\; 33} - {P\; 43}}}} ) \times \frac{1}{16}}} & ( {{Eq}.\mspace{14mu} 21} )\end{matrix}$

A correlation value in the horizontal direction is calculated from Eq.22. FIG. 14 is a view showing a correlation value calculation method inthe horizontal direction.

$\begin{matrix}{{Ch\_ m} = {( {{{{P\; 20} - {P\; 21}}} + {{{P\; 21} - {P\; 22}}} + {{{P\; 22} - {P\; 23}}} + {{{P\; 23} - {P\; 24}}} + {{{P\; 10} - {P\; 11}}} + {{{P\; 11} - {P\; 12}}} + {{{P\; 12} - {P\; 13}}} + {{{P\; 13} - {P\; 14}}} + {{{P\; 30} - {P\; 31}}} + {{{P\; 31} - {P\; 32}}} + {{{P\; 32} - {P\; 33}}} + {{{P\; 33} - {P\; 34}}}} ) \times \frac{1}{6}}} & ( {{Eq}.\mspace{14mu} 22} )\end{matrix}$

A correlation value in the diagonal A direction is calculated from Eq.23. FIG. 15 is a view showing a correlation value calculation method inthe diagonal A direction.

$\begin{matrix}{{{Cd}_{A}{\_ m}} = {( {{{{P\; 00} - {P\; 11}}} + {{{P\; 11} - {P\; 22}}} + {{{P\; 22} - {P\; 33}}} + {{{P\; 33} - {P\; 44}}} + {{{P\; 10} - {P\; 21}}} + {{{P\; 21} - {P\; 32}}} + {{{P\; 32} - {P\; 43}}} + {{{P\; 01} - {P\; 12}}} + {{{P\; 12} - {P\; 23}}} + {{{P\; 23} - {P\; 34}}}} ) \times \frac{1}{5}}} & ( {{Eq}.\mspace{14mu} 23} )\end{matrix}$

A correlation value in the diagonal B direction is calculated from Eq.24. FIG. 16 is a view showing a correlation value calculation method inthe diagonal B direction.

$\begin{matrix}{{{Cd}_{B}{\_ m}} = {( {{{{P\; 04} - {P\; 13}}} + {{{P\; 13} - {P\; 22}}} + {{{P\; 22} - {P\; 31}}} + {{{P\; 31} - {P\; 40}}} + {{{P\; 03} - {P\; 12}}} + {{{P\; 12} - {P\; 21}}} + {{{P\; 21} - {P\; 30}}} + {{{P\; 14} - {P\; 23}}} + {{{P\; 23} - {P\; 32}}} + {{{P\; 32} - {P\; 41}}}} ) \times \frac{1}{5}}} & ( {{Eq}.\mspace{14mu} 24} )\end{matrix}$

Further, the distance between pixels used for calculation of thedifferential value in FIGS. 13 and 14 is different from that in FIGS. 15and 16. But, herein, the differential value is not multiplied by acoefficient taking the distance between pixels into account as discussedin Eq. 15. The reason is that the difference in the distance betweenpixels is not much large, but the pixel differential value in Eqs. 21and 22, for example, may be multiplied by the square root of 2.

Further, in Eqs. 21 to 24, for easier comparison with theabove-discussed correlation value for color image, the scale is matched.Specifically, the distance between pixels used for calculation shown inFIGS. 13 to 16 is a distance between two adjacent pixels. Therefore, inEqs. 21 to 24, as the result that each pixel differential value ismultiplied by 2 to match the scale, the final multiplier (⅙ and ⅕) istwice the reciprocal of the cumulative number in each equation. Sincethe correlation direction in a gray image, however, is judged by usingonly the correlation value for gray image, it is not always necessary tomatch the scale.

<5. Selection of Correlation Judgment Method and Pixel InterpolationMethod>

The selection circuit 33 selects a correlation judgment method and apixel interpolation method on the basis of the relation among the chromafactor K_(L) calculated by the chroma value calculation circuit 31 andthreshold values TH1 and TH2 (TH1≧TH2). Specifically, selection of thecorrelation judgment method is to select among methods of judging thecorrelation direction, by adopting the correlation value for gray image,by adopting the correlation value for color image or by performing anoverall judgment on the correlation values for gray image and colorimage to select one and using the selected correlation value. Selectionof the pixel interpolation method is to select whether the pixelinterpolation method for gray image or the method for color image isadopted.

FIG. 17 is a view showing the types of correlation judgment method andpixel interpolation method to be selected on the basis of the relationamong the chroma factor K_(L) and the threshold values TH1 and TH2.Specifically, selections are classified into the following patterns (a)to (c) of combinations of the methods.

(a) K_(L)>TH1

correlation judgment method: the correlation direction is judged byusing the correlation value for color image.

pixel interpolation method: the pixel interpolation method for colorimage is used.

(b) TH1≧K_(L)>TH2

correlation judgment method: the correlation direction is judged byusing the correlation value selected by overall judgment on thecorrelation values for color image and gray image.

pixel interpolation method: the pixel interpolation method for colorimage is used.

(c) TH2≧K_(L)

correlation judgment method: the correlation direction is judged byusing the correlation value for gray image.

pixel interpolation method: the pixel interpolation method for grayimage is used.

By using Eqs. 13 to 20, the correlation values Cv_c, Ch_c, Cd_(A) _(—) cand Cd_(B) _(—) c for color image in the four directions are calculated.Further, by using Eqs. 21 to 24, the correlation values Cv_m, Ch_m,Cd_(A) _(—) m and Cd_(B) _(—) m for gray image in the four directionsare calculated. The selection circuit 33 selects judgment correlationvalues Cv, Ch, Cd_(A) and Cd_(B) which are to be used actually forjudgment on the correlation direction, out of the calculated correlationvalues for gray image and color image as discussed below.

(5-1) (a) Judgment Correlation Value when K_(L)>TH1

As shown in Eq. 25, the correlation values for color image are used asthe judgment correlation values Cv, Ch, Cd_(A) and Cd_(B).Cv=Cv _(—) cCh=Ch _(—) cCd _(A) =Cd _(A) _(—) cCd _(B) =Cd _(B) _(—) c  (Eq. 25)

(5-2) (c) Judgment Correlation Value when TH2≧K_(L)

As shown in Eq. 26, the correlation values for gray image are used asthe judgment correlation values Cv, Ch, Cd_(A) and Cd_(B).Cv=Cv _(—) mCh=Ch _(—) mCd _(A) =Cd _(A) _(—) mCd _(B) =Cd _(B) _(—) m  (Eq. 26)

(5-3) (b) Judgment Correlation Value when TH1≧K_(L)>TH2

By overall judgment on the correlation values for gray image and colorimage, the judgment correlation values Cv, Ch, Cd_(A) and Cd_(B) aredetermined. Detailed discussion will be made below on this judgmentmethod.

First, as shown in Eq. 27, a differential absolute value d_Cv_c betweenthe correlation values Cv_c and Ch_c for color image is obtained.d _(—) Cv _(—) c=abs(Cv _(—) c−Ch _(—) c)  (Eq. 27)

As shown in Eq. 28, a differential absolute value d_Cv_m between thecorrelation values Cv_m and Ch_m for gray image is obtained.d _(—) Cv _(—) m=abs(Cv _(—) m−Ch _(—) m)  (Eq. 28)

Further, as shown in Eq. 29, a differential absolute value d_Cv betweenthe differential absolute values d_Cv_c and d_Cv_m is compared with athreshold value THv.abs(d _(—) Cv _(—) c−d _(—) Cv _(—) m)=d _(—) Cv≦THv  (Eq. 29)

If the relation between the differential absolute value d_Cv and thethreshold value THv satisfies Eq. 29, further judgment is made onwhether Eq. 30 is satisfied or not.Cv _(—) c<Cv _(—) m  (Eq. 30)

If Eq. 30 is satisfied, the correlation value Cv_c for color image isadopted as the judgment correlation value Cv. Specifically, Cv=Cv_c.

If Eq. 30 is not satisfied, the correlation value Cv_m for gray image isadopted as the judgment correlation value Cv. Specifically, Cv=Cv_m.

If the relation between the differential absolute value d_Cv and thethreshold value THv satisfies Eq. 29, further judgment is made onwhether Eq. 31 is satisfied or not.Ch _(—) c<Ch _(—) m  (Eq. 31)

If Eq. 31 is satisfied, the correlation value Ch_c for color image isadopted as the judgment correlation value Ch. Specifically, Ch=Ch_c.

If Eq. 31 is not satisfied, the correlation value Ch_m for gray image isadopted as the judgment correlation value Ch. Specifically, Ch=Ch_m.

Thus, when the differential absolute value d_Cv is smaller than thethreshold value THv, the difference between the differential absolutevalue d_Cv_c and the differential absolute value d_Cv_m is small. Inother words, it is not assumed that either the vertical direction or thehorizontal direction may have a strong correlation. In such a case, ineach of the vertical direction and the horizontal direction, thecorrelation values for gray image and color image are compared with eachother, and the smaller one, i.e., one having high correlation isselected.

If the relation between the differential absolute value d_Cv and thethreshold value THv does not satisfy Eq. 29, further judgment is made onwhether Eq. 32 is satisfied or not.d _(—) Cv _(—) c>d _(—) Cv _(—) m  (Eq. 32)

If Eq. 32 is satisfied, the correlation values Cv_c and Ch_c for colorimage are adopted for correlation judgment. Specifically, Cv=Cv_c andCh=Ch_c.

If Eq. 32 is not satisfied, the correlation values Cv_m and Ch_m forgray image are adopted for correlation judgment. Specifically, Cv=Cv_mand Ch=Ch_m.

Thus, when the differential absolute value d_Cv is larger than thethreshold value THv, the difference between the differential absolutevalue d_Cv_c and d_Cv_m is large. In other words, it is assumed thateither the vertical direction or the horizontal direction may have astrong correlation. In such a case, the differential absolute valuesd_Cv_c and d_Cv_m are compared with each other, and the correlationvalue for the image having the larger differential absolute value isselected.

Subsequently, as shown in Eq. 33, a differential absolute value d_Cdg_cbetween the correlation values Cd_(A) _(—) c and Cd_(B) _(—) c for colorimage is obtained.d _(—) Cdg _(—) c=abs(Cd _(A) _(—) c−Cd _(B) −c)  (Eq. 33)

Further, as shown in Eq. 34, a differential absolute value d_Cdg_mbetween the correlation values Cd_(A) _(—) m and Cd_(B) _(—) m for grayimage is obtained.d _(—) Cdg _(—) m=abs(Cd _(A) _(—) m−Cd _(B) −m)  (Eq. 34)

Furthermore, as shown in Eq. 35, a differential absolute value d_Cdgbetween the differential absolute values d_Cdg_c and d_Cdg_m is comparedwith a threshold value THdg.abs(d _(—) Cdg _(—) c−d _(—) Cdg _(—) m)=d _(—) Cdg≦THdg  (Eq. 35)

If the relation between the differential absolute value d_Cdg and thethreshold value THdg satisfies Eq. 35, further judgment is made onwhether Eq. 36 is satisfied or not.Cd _(A) _(—) c<Cd _(A) _(—) m  (Eq. 36)

If Eq. 36 is satisfied, the correlation value Cd_(A) _(—) c for colorimage is adopted as the judgment correlation value Cd_(A). Specifically,Cd_(A)=Cd_(A) _(—) c.

If Eq. 36 is not satisfied, the correlation value Cd_(A) _(—) m for grayimage is adopted as the judgment correlation value Cd_(A). Specifically,Cd_(A)=Cd_(A) _(—) m.

If the relation between the differential absolute value d_Cdg and thethreshold value THdg satisfies Eq. 35, further judgment is made onwhether Eq. 37 is satisfied or not.Cd _(B) _(—) c<Cd _(B) _(—) m  (Eq. 37)

If Eq. 37 is satisfied, the correlation value Cd_(B) _(—) c for colorimage is adopted as the judgment correlation value Cd_(B). Specifically,Cd_(B)=Cd_(B) _(—) c.

If Eq. 37 is not satisfied, the correlation value Cd_(B) _(—) m for grayimage is adopted as the judgment correlation value Cd_(B). Specifically,Cd_(B)=Cd_(B) _(—) m.

Thus, when the differential absolute value d_Cdg is smaller than thethreshold value THdg, the difference between the differential absolutevalue d_Cdg_c and d_Cdg_m is small. In other words, it is not assumedthat either the diagonal A direction or the diagonal B direction mayhave a strong correlation. In such a case, in each of the diagonal Adirection and the diagonal B direction, the correlation values for grayimage and color image are compared with each other, and the smaller one,i.e., one having high correlation is selected.

If the relation between the differential absolute value d_Cdg and thethreshold value THdg does not satisfy Eq. 35, further judgment is madeon whether Eq. 38 is satisfied or not.d _(—) Cdg _(—) c>d _(—) Cdg _(—) m  (Eq. 38)

If Eq. 38 is satisfied, the correlation values Cd_(A) _(—) c and Cd_(B)_(—) c for color image are adopted for correlation judgment.Specifically, Cd_(A)=Cd_(A) _(—) c and Cd_(B)=Cd_(B) _(—) c

If Eq. 38 is not satisfied, the correlation values Cd_(A) _(—) m andCd_(B) _(—) m for gray image are adopted for correlation judgment.Specifically, Cd_(A)=Cd_(A) _(—) m and Cd_(B)=Cd_(B) _(—) m.

Thus, when the differential absolute value d_Cdg is larger than thethreshold value THdg, the difference between the differential absolutevalue d_Cdg_c and d_Cdg_m is large. In other words, it is assumed thateither the diagonal A direction or the diagonal B direction may have astrong correlation. In such a case, the differential absolute valuesd_Cdg_c and d_Cdg_m are compared with each other, and the correlationvalue for the image having the larger differential absolute value isselected.

In the case of (b) where TH1≧K_(L)>TH2, through the above operation, thejudgment correlation values Cv, Ch, Cd_(A) and Cd_(B) are selected byoverall judgment on the correlation values for gray image and colorimage.

The selection circuit 33 performs the above computation, to select thejudgment correlation values Cv, Ch, Cd_(A) and Cd_(B) in each of thecases (a), (b) and (c).

Thus, in the first preferred embodiment, the combinations of thecorrelation judgment method and the pixel interpolation method areclassified into three patterns from the relation among the chroma factorK_(L) and the threshold values TH1 and TH2. Specifically, instead ofproviding one threshold value and making a judgment on whether the grayimage or the color image, two threshold values TH1 and TH2 are providedand a boundary area between the gray image and the color image isthereby smoothed. Particularly, in an image existing near the boundarybetween a gray image and a color image, it thereby becomes possible toease the visual unpleasantness after interpolation.

More specifically, the image existing near the boundary between a grayimage and a color image has almost equal values of RGB components butthere is a little variation in these values. Therefore, for judgment oncorrelation, paying attention to that the variation in RGB components issmall, the correlation value is calculated by using pixels which are asnear as possible without distinction of RGB. Alternatively, payingattention to that there is some variation in RGB components, thecorrelation value is calculated with distinction of RGB. Through overalljudgment over such two ideas to select an optimum correlation value, theaccuracy in judgment on the correlation direction is improved. In thiscase, if the variation in RGB components is disregarded and the imagenear the boundary is regarded as a gray image to perform pixelinterpolation, there arises a possibility that false colors may occur.Then, as to pixel interpolation, a pixel interpolation process for colorimage is performed.

Further, in the first preferred embodiment, the chroma factor K_(L)obtained by normalizing the chroma evaluation value L is used and thechroma factor K_(L) and the threshold values TH1 and TH2 are comparedwith one another to judge whether a gray image or a color image. But,this is done for convenience in operation, and substantially, the chromaevaluation value L and the two threshold values are compared with oneanother, to judge whether a gray image or a color image. After selectingthe correlation judgment method and the pixel interpolation method, theselection circuit 33 gives information on this selection to the firstcorrelation judgment circuit 341 and the second correlation judgmentcircuit 342. The selection information includes information indicatingwhether the correlation judgment method and the pixel interpolationmethod are for gray image or color image and information indicating theselected judgment correlation values Cv, Ch, Cd_(A) and Cd_(B).

<6. Judgment on Correlation Direction of Each Pixel>

As discussed above, after selecting the judgment correlation values Cv,Ch, Cd_(A) and Cd_(B), the selection circuit 33 outputs the pixel signaland the selection information including the information on the judgmentcorrelation values to the first correlation judgment circuit 341 and thesecond correlation judgment circuit 342. Specifically, the judgmentcorrelation values Cv, Ch, Cd_(A) and Cd_(B) calculated by the selectioncircuit 33 are outputted to both the first correlation judgment circuit341 and the second correlation judgment circuit 342 and the pixel signalinputted from the signal processing circuit 2 is also outputted to boththe first correlation judgment circuit 341 and the second correlationjudgment circuit 342. The first correlation judgment circuit 341 and thesecond correlation judgment circuit 342 are processing parts to judgethe correlation with respect to the specified pixel on the basis of thejudgment correlation values Cv, Ch, Cd_(A) and Cd_(B). The firstcorrelation judgment circuit 341 judges the correlation direction, withthe correlation with respect to the pixel signal evaluated highly. Thesecond correlation judgment circuit 342 judges the correlationdirection, with the correlation with respect to the pixel signalevaluated low, as compared with the first correlation judgment circuit341.

FIG. 18 is a graph showing a correspondence of correlation values whichthe first correlation judgment circuit 341 uses for judgment on thecorrelation direction. The vertical axis represents the judgmentcorrelation value Cv and the horizontal axis represents the judgmentcorrelation value Ch.

When the relation between the judgment correlation value Cv and thejudgment correlation value Ch is found in an area A1, the firstcorrelation judgment circuit 341 judges that the correlation directionof the specified pixel is the horizontal direction. When the relationbetween the judgment correlation value Cv and the judgment correlationvalue Ch is found in an area A2, the first correlation judgment circuit341 judges that the correlation direction of the specified pixel is thevertical direction. When the relation between the judgment correlationvalue Cv and the judgment correlation value Ch is found in an area A3,the first correlation judgment circuit 341 judges that there is nocorrelation of the specified pixel in any direction. When the relationbetween the judgment correlation value Cv and the judgment correlationvalue Ch is found in an area A4, the first correlation judgment circuit341 judges that the correlation of the specified pixel is high in boththe vertical and horizontal directions.

The first correlation judgment circuit 341 uses the correspondence viewshown in FIG. 19, together with that of FIG. 18. The correspondence viewof FIG. 19 shows a correspondence between the judgment correlationvalues Cd_(A) and Cd_(B) and the correlation direction. The verticalaxis of FIG. 19 represents the judgment correlation value Cd_(A) and thehorizontal axis represents the judgment correlation value Cd_(B). Thearea B1 is an area for judgment that the correlation direction is thediagonal B direction, and the area B2 is an area for judgment that thecorrelation direction is the diagonal A direction. Further, the area B3is an area for judgment that there is no correlation in any direction,and the area B4 is an area for judgment that the correlation is high inboth the diagonal A direction and the diagonal B direction.

The first correlation judgment circuit 341 compares the four judgmentcorrelation values Cv, Ch, Cd_(A) and Cd_(B). Then, when the judgmentcorrelation value Cv or the judgment correlation value Ch is smallest,the correspondence view of FIG. 18 is used. Then, the first correlationjudgment circuit 341 determines the correlation direction, depending onwhich of the areas A1 to A4 where the correspondence of the judgmentcorrelation values is found. On the other hand, when the judgmentcorrelation value Cd_(A) or the judgment correlation value Cd_(B) issmallest, the correspondence view of FIG. 19 is used. Then, the firstcorrelation judgment circuit 341 determines the correlation direction,depending on which of the areas B1 to B4 where the correspondence of thejudgment correlation values is found.

After the correlation direction is determined, as discussed later, thefirst interpolation circuit 351 performs the pixel interpolation processby using the pixels in the correlation direction. Specifically, when thecorrespondence of the judgment correlation values is found in the areaA1, the pixel interpolation is performed by using the pixels in thehorizontal direction. When the correspondence of the judgmentcorrelation values is found in the area A2, the pixel interpolation isperformed by using the pixels in the vertical direction. When thecorrespondence of the judgment correlation values is found in the areaB1, the pixel interpolation is performed by using the pixels in thediagonal B direction. When the correspondence of the judgmentcorrelation values is found in the area B2, the pixel interpolation isperformed by using the pixels in the diagonal A direction. Further, whenthe correspondence of the judgment correlation values is found in thearea A3 or B3, for example, median interpolation is performed. When thecorrespondence of the judgment correlation values is found in the areaA4 or B4, mean value interpolation is performed.

On the other hand, FIG. 20 is a graph showing a correspondence ofjudgment correlation values which the second correlation judgmentcircuit 342 uses for judgment on the correlation direction. The verticalaxis represents the judgment correlation value Cv and the horizontalaxis represents the judgment correlation value Ch.

When the relation between the judgment correlation value Cv and thejudgment correlation value Ch is found in an area A5, the secondcorrelation judgment circuit 342 judges that the correlation directionof the specified pixel is the horizontal direction. When the relationbetween the judgment correlation value Cv and the judgment correlationvalue Ch is found in an area A6, the second correlation judgment circuit342 judges that the correlation direction of the specified pixel is thevertical direction. When the relation between the judgment correlationvalue Cv and the judgment correlation value Ch is found in an area A7,the second correlation judgment circuit 342 judges that there is nocorrelation of the specified pixel in any direction. When the relationbetween the judgment correlation value Cv and the judgment correlationvalue Ch is found in an area A8, the second correlation judgment circuit342 judges that the correlation of the specified pixel is high in boththe vertical and horizontal directions.

The second correlation judgment circuit 342 uses the correspondence viewshown in FIG. 21, together with that of FIG. 20. The correspondence viewof FIG. 21 shows a correspondence between the judgment correlationvalues Cd_(A) and Cd_(B) and the correlation direction. The verticalaxis of FIG. 21 represents the judgment correlation value Cd_(A) and thehorizontal axis represents the judgment correlation value Cd_(B). Thearea B5 is an area for judgment that the correlation direction is thediagonal B direction, and the area B6 is an area for judgment that thecorrelation direction is the diagonal A direction. Further, the area B7is an area for judgment that there is no correlation in any direction,and the area B8 is an area for judgment that the correlation is high inboth the diagonal A direction and the diagonal B direction.

The second correlation judgment circuit 342 compares the four judgmentcorrelation values Cv, Ch, Cd_(A) and Cd_(B). Then, when the judgmentcorrelation value Cv or the judgment correlation value Ch is smallest,the correspondence view of FIG. 20 is used. Then, the second correlationjudgment circuit 342 determines the correlation direction, depending onwhich of the areas A5 to A8 where the correspondence of the judgmentcorrelation values is found. On the other hand, when the judgmentcorrelation value Cd_(A) or the judgment correlation value Cd_(B) issmallest, the correspondence view of FIG. 21 is used. Then, the secondcorrelation judgment circuit 342 determines the correlation direction,depending on which of the areas B5 to B8 where the correspondence of thejudgment correlation values is found.

After the correlation direction is determined, as discussed later, thesecond interpolation circuit 352 performs the pixel interpolationprocess by using the pixels in the correlation direction. Specifically,when the correspondence of the judgment correlation values is found inthe area A5, the pixel interpolation is performed by using the pixels inthe horizontal direction. When the correspondence of the judgmentcorrelation values is found in the area A6, the pixel interpolation isperformed by using the pixels in the vertical direction. When thecorrespondence of the judgment correlation values is found in the areaB5, the pixel interpolation is performed by using the pixels in thediagonal B direction. When the correspondence of the judgmentcorrelation values is found in the area B6, the pixel interpolation isperformed by using the pixels in the diagonal A direction. Further, whenthe correspondence of the judgment correlation values is found in thearea A7 or B7, for example, the median interpolation is performed. Whenthe correspondence of the judgment correlation values is found in thearea A8 or B8, the mean value interpolation is performed.

Thus, the first correlation judgment circuit 341 and the secondcorrelation judgment circuit 342 determine the correlation direction byusing the correspondence of the judgment correlation values shown inFIGS. 18 to 21. As a result, the first correlation judgment circuit 341determines the correlation direction, with the correlation between thespecified pixel and its surrounding pixels evaluated highly. In otherwords, the first correlation judgment circuit 341 actively uses thepixels in the correlation direction, to perform interpolation. On theother hand, the second correlation judgment circuit 342 determines thecorrelation direction, with the correlation between the specified pixeland its surrounding pixels evaluated low, as compared with the firstcorrelation judgment circuit 341. In other words, the secondinterpolation circuit 352 is an interpolation circuit which activelyadopts the median interpolation and the mean value interpolation.

Both the area A1 in FIG. 18 and the area A5 in FIG. 20 are areas wherethe correlation is judged to be high in the horizontal direction. As canbe seen from the comparison between FIGS. 18 and 20, the inclination ofthe line F4 defining the area A5 is larger than that of the line F1defining the area A1. Further, the value of intersection point betweenthe line F4 and the vertical axis is larger than that between the lineF1 and the vertical axis. In other words, when there is a relation thatthe judgment correlation value Ch is slightly smaller than the judgmentcorrelation value Cv, the first correlation judgment circuit 341actively adopts the relation to judge that the correlation in thehorizontal direction is high. On the other hand, when there is arelation that the judgment correlation value Ch is sufficiently smallerthan the judgment correlation value Cv, the second correlation judgmentcircuit 342 judges that the correlation in the horizontal direction ishigh.

Further, the inclination of the line F5 defining the area A6 is smallerthan that of the line F2 defining the area A2. Furthermore, the value ofintersection point between the line F5 and the horizontal axis is largerthan that between the line F2 and the horizontal axis. In other words,when there is a relation that the judgment correlation value Cv isslightly smaller than the judgment correlation value Ch, the firstcorrelation judgment circuit 341 actively adopts the relation to judgethat the correlation in the vertical direction is high. On the otherhand, when there is a relation that the judgment correlation value Cv issufficiently smaller than the judgment correlation value Ch, the secondcorrelation judgment circuit 342 judges that the correlation in thevertical direction is high.

As to the line F3 defining the areas A3 and A4 and the line F6 definingthe areas A7 and A8, the relation shown in FIGS. 18 and 20 is only oneexample. In other words, though the values of intersection pointsbetween the line F6 and the axes are larger those between the line F3and the axes, the relation between them is not limited to such arelation as above.

The relation between FIGS. 19 and 21 is the same as above. Theinclination of the line F14 defining the area B5 is larger than that ofthe line F11 defining the area B1. Further, the value of intersectionpoint between the line F14 and the vertical axis is larger than thatbetween the line F11 and the vertical axis. When there is a relationthat the judgment correlation value Cd_(B) is slightly smaller than thejudgment correlation value Cd_(A), the first correlation judgmentcircuit 341 actively adopts the relation to judge that the correlationin the diagonal B direction is high. On the other hand, when there is arelation that the judgment correlation value Cd_(B) is sufficientlysmaller than the judgment correlation value Cd_(A), the secondcorrelation judgment circuit 342 judges that the correlation in thediagonal B direction is high.

Further, the inclination of the line F15 defining the area B6 is smallerthan that of the line F12 defining the area B2. Furthermore, the valueof intersection point between the line F15 and the horizontal axis islarger than that between the line F12 and the horizontal axis. In otherwords, when there is a relation that the judgment correlation valueCd_(A) is slightly smaller than the judgment correlation value Cd_(B),the first correlation judgment circuit 341 actively adopts the relationto judge that the correlation in the diagonal A direction is high. Onthe other hand, when there is a relation that the judgment correlationvalue Cd_(A) is sufficiently smaller than the judgment correlation valueCd_(B), the second correlation judgment circuit 342 judges that thecorrelation in the diagonal A direction is high.

As to the line F13 defining the areas B3 and B4 and the line F16defining the areas B7 and B8, the relation shown in FIGS. 19 and 21 isonly one example. In other words, though the values of intersectionpoints between the line F16 and the axes are larger those between theline F13 and the axes, the relation between them is not limited to sucha relation as above.

<7. Pixel Interpolation Process>

Now, discussion will be made on the pixel interpolation processperformed by the first interpolation circuit 351 and the secondinterpolation circuit 352. The first interpolation circuit 351 and thesecond interpolation circuit 352, as discussed above, perform the pixelinterpolation process on the correlation direction determined by thefirst correlation judgment circuit 341 and the second correlationjudgment circuit 342. Then, the first interpolation circuit 351 and thesecond interpolation circuit 352 perform the pixel interpolation processfor either gray image or color image on the basis of the selectioninformation outputted from the selection circuit 33. Specifically, ifthe above-discussed pattern (c) is selected in the selection circuit 33,the pixel interpolation process for gray image is performed, and if theabove pattern (a) or (b) is selected in the selection circuit 33, thepixel interpolation process for color image is performed (see <5.Selection of Correlation Judgment Method and Pixel InterpolationMethod>.

(7-1) Pixel Interpolation for Gray Image

If the above-discussed pattern (c) is selected in the selection circuit33, the pixel interpolation process for gray image is performed on thecorrelation direction determined in the first correlation judgmentcircuit 341 and the second correlation judgment circuit 342. In thepixel interpolation for gray image, without distinction of the colorcomponent of the specified pixel among R, G and B, the pixelinterpolation process is performed by using the pixels existing in thedetermined correlation direction. In other words, without considerationof what color component among R, G and B the specified pixel is and whatcolor component among R, G and B its surrounding pixels are, thespecified pixel is interpolated by using its surrounding pixels.

Specifically, if the correlation value is judged to be high in thevertical direction, in other words, if the first correlation judgmentcircuit 341 judges that the correlation direction belongs to the areaA2, the first interpolation circuit 351 performs the pixel interpolationprocess by using Eq. 39. Alternatively, if the second correlationjudgment circuit 342 judges that the correlation direction belongs tothe area A6, the second interpolation circuit 352 performs the pixelinterpolation process by using Eq. 39.

$\begin{matrix}{S_{out} = \frac{{P\; 12} + {2 \times P\; 22} + {P\; 32}}{4}} & ( {{Eq}.\mspace{14mu} 39} )\end{matrix}$

Further, if the correlation value is judged to be high in the horizontaldirection, in other words, if the first correlation judgment circuit 341judges that the correlation direction belongs to the area A1, the firstinterpolation circuit 351 performs the pixel interpolation process byusing Eq. 40. Alternatively, if the second correlation judgment circuit342 judges that the correlation direction belongs to the area A5, thesecond interpolation circuit 352 performs the pixel interpolationprocess by using Eq. 40.

$\begin{matrix}{S_{out} = \frac{{P\; 21} + {2 \times P\; 22} + {P\; 23}}{4}} & ( {{Eq}.\mspace{20mu} 40} )\end{matrix}$

Furthermore, if the correlation value is judged to be high in thediagonal A direction, in other words, if the first correlation judgmentcircuit 341 judges that the correlation direction belongs to the areaB2, the first interpolation circuit 351 performs the pixel interpolationprocess by using Eq. 41. Alternatively, if the second correlationjudgment circuit 342 judges that the correlation direction belongs tothe area B6, the second interpolation circuit 352 performs the pixelinterpolation process by using Eq. 41.

$\begin{matrix}{S_{out} = \frac{{P\; 11} + {2 \times P\; 22} + {P\; 33}}{4}} & ( {{Eq}.\mspace{14mu} 41} )\end{matrix}$

Still further, if the correlation value is judged to be high in thediagonal B direction, in other words, if the first correlation judgmentcircuit 341 judges that the correlation direction belongs to the areaB1, the first interpolation circuit 351 performs the pixel interpolationprocess by using Eq. 42. Alternatively, if the second correlationjudgment circuit 342 judges that the correlation direction belongs tothe area B5, the second interpolation circuit 352 performs the pixelinterpolation process by using Eq. 42.

$\begin{matrix}{S_{out} = \frac{{P\; 13} + {2 \times P\; 22} + {P\; 31}}{4}} & ( {{Eq}.\mspace{14mu} 42} )\end{matrix}$

In Eqs. 39 to 42, the term of P22 is multiplied by the coefficient of“2” and this is for weighting in accordance with the distance from thespecified pixel. Further, if it is judged that the correlation is highin all the directions (the area A4, B4, A8 or B8), for example, the meanvalue interpolation is performed. If it is judged that there is nocorrelation in any direction (the area A3, B3, A7 or B7), for example,the median interpolation is performed.

(7-2) Pixel Interpolation for Color Image

If the above-discussed pattern (a) or (b) is selected in the selectioncircuit 33, the first interpolation circuit 351 and the secondinterpolation circuit 352 perform the pixel interpolation process forcolor image on the correlation direction determined in the firstcorrelation judgment circuit 341 and the second correlation judgmentcircuit 342. In the pixel interpolation for color image, aninterpolation computation method changes depending on the colorcomponent of the specified pixel among R, G and B. Specifically, thepixel interpolation process is performed by using the pixels which existin the correlation direction determined by the first correlationjudgment circuit 341 and the second correlation judgment circuit 342 andhave the same color as the color of the pixel to be interpolated.

For example, if the specified pixel is a G pixel and the correlation inthe vertical direction is judged to be high (the area A2 or A6), the Rcomponent and the B component of the specified pixel are interpolated byusing the R pixels and the B pixels existing in the vertical direction.Further, if the specified pixel is a G pixel and the correlation in thehorizontal direction is judged to be high (the area A1 or A5), the Bcomponent and the R component of the specified pixel are interpolated byusing the B pixels and the R pixels existing in the horizontaldirection.

If the specified pixel is a G pixel and the correlation in the diagonalA direction is judged to be high (the area B2 or B6), the R componentand the B component of the specified pixel are interpolated by using theR pixels and the B pixels existing in the diagonal A direction. Further,if the specified pixel is a G pixel and the correlation in the diagonalB direction is judged to be high (the area B1 or B5), the B componentand the R component of the specified pixel are interpolated by using theB pixels and the R pixels existing in the diagonal B direction.

If there is a pixel having the color component to be interpolated on theline in a direction for interpolation, the pixel interpolation processcan be performed by calculating an average value of the pixels havingthe same color which exist on the line or performing linearinterpolation. Depending on the pixel array, however, there is sometimesno pixel having the color component to be interpolated on the line inthe direction for the interpolation. In such a case, a method in which apixel value of the pixel to be interpolated is estimated from the rateof pixel change (Laplacian) in a direction orthogonal to the line in thedirection for the interpolation may be used.

Thus, the pixel interpolation process for color image is performed byusing the pixels existing in the correlation direction, which have thesame color as the color of the pixel to be interpolated, to interpolatethe specified pixel. Alternatively, if there is no pixel having the samecolor as the color of the pixel to be interpolated in the correlationdirection, interpolation is performed on the specified pixel by usingthe estimated value as a pixel value of the pixel having the same colorin the correlation direction.

As discussed above, the image processing circuit 3 of the firstpreferred embodiment obtains the chroma evaluation value L of the matrixarea and selects the correlation judgment method and the pixelinterpolation method on the specified pixel in the matrix area on thebasis of the chroma evaluation value L. Then, for judging whether theprocess for gray image or for color image on the basis of the chromaevaluation value L, the two threshold values TH1 and TH2 are used. Forthe area existing at the boundary between a gray image and a colorimage, the correlation direction is judged by using the correlationvalue selected by overall judgment on the correlation values for grayimage and color image, and by using the pixel interpolation method forcolor image, it is possible to reduce false colors and improve the senseof resolution.

<8. Color Space Conversion Process>

After performing the pixel interpolation process on each pixel, thefirst interpolation circuit 351 outputs a complete pixel signal afterinterpolation to the first color space conversion circuit 361. In otherwords, as to the signal inputted to the first color space conversioncircuit 361, each pixel includes signals for all the color components ofRGB. Further, after performing the pixel interpolation process on eachpixel, the second interpolation circuit 352 outputs a complete pixelsignal after interpolation to the second color space conversion circuit362. In other words, as to the signal inputted to the second color spaceconversion circuit 362, each pixel includes signals for all the colorcomponents of RGB.

Then, the first color space conversion circuit 361 generates a luminancesignal (Y signal) from the pixel signal of RGB for each pixel. On theother hand, the second color space conversion circuit 362 generatescolor difference signals (Cb and Cr signals) from the pixel signal ofRGB for each pixel. Thus, the RGB signal of Bayer array outputted fromthe image pickup element 1 is converted into the luminance signal (Ysignal) and the color difference signals (Cb and Cr signals).

As discussed above, the luminance signal outputted from the first colorspace conversion circuit 361 is a signal generated from the RGB signalwhich is interpolated by the first interpolation circuit 351. The RGBsignal interpolated by the first interpolation circuit 351 is a signalwhich is subjected to the pixel interpolation with the correlationevaluated highly, i.e., a signal maintaining high resolution. It isthereby possible to keep the sense of resolution of the generated YUVsignal high.

On the other hand, the color difference signals outputted from thesecond color space conversion circuit 362 are signals generated from theRGB signal which is interpolated by the second interpolation circuit352. The RGB signal interpolated by the second interpolation circuit 352is a signal which is subjected to the pixel interpolation with thecorrelation evaluated relatively low, i.e., a signal whose noise issuppressed. In other words, this is a signal to which an LPF (Low PassFilter) is applied. It is thereby possible to suppress the noise of thegenerated YUV signal even if a RAW image having high noise is outputtedfrom the image pickup element 1.

The luminance signal (Y signal) outputted from the first color spaceconversion circuit 361 and the color difference signals (Cb and Crsignals) outputted from the second color space conversion circuit 362are stored into the memory 5.

The Second Preferred Embodiment

FIG. 22 is a view showing a criterion of judgment on the correlationjudgment method and the pixel interpolation method in the secondpreferred embodiment. In the first preferred embodiment, as shown inFIG. 17, the selection circuit 33 selects the correlation judgmentmethod and the pixel interpolation method by using the two thresholdvalues TH1 and TH2. In the second preferred embodiment, as shown in FIG.22, the selection circuit 33 selects the correlation judgment method andthe pixel interpolation method by using three threshold values TH1, THMand TH2 (TH1≧THM≧TH2), as discussed below.

(a′) K_(L)>TH1

correlation judgment method: the correlation direction is judged byusing the correlation value for color image.

pixel interpolation method: the pixel interpolation method for colorimage is used.

(b′) TH1≧K_(L)>THM

correlation judgment method: the correlation direction is judged byusing the correlation value selected by overall judgment on thecorrelation values for color image and gray image.

pixel interpolation method: the pixel interpolation method for colorimage is used.

(b″) THM≧K_(L)>TH2

correlation judgment method: the correlation direction is judged byusing the correlation value for gray image.

pixel interpolation method: the pixel interpolation method for colorimage is used.

(c) TH2≧K_(L)

correlation judgment method: the correlation direction is judged byusing the correlation value for gray image.

pixel interpolation method: the pixel interpolation method for grayimage is used.

Thus, the selection circuit 33 uses the three threshold values to selectthe correlation judgment method and the pixel interpolation method infour patterns. The pattern (b) in the first preferred embodiment isfurther classified into two patterns (b′) and (b″). The pattern (b′) isthe same as the pattern (b) in the first preferred embodiment. With thepattern (b″), on the area closer to a gray area, it becomes possible tojudge the correlation direction with high precision through correlationjudgment without distinction of RGB. This threshold value THM may be setfreely from the threshold value TH2 to the threshold value TH1. The caseof the first preferred embodiment is a case where the threshold valuesTHM and TH2 coincide with each other.

While the invention has been shown and described in detail, theforegoing description is in all aspects illustrative and notrestrictive. It is therefore understood that numerous modifications andvariations can be devised without departing from the scope of theinvention.

1. A pixel interpolation method comprising the steps of: (a) inputting apixel signal of a predetermined color space; (b) calculating a chromaevaluation value of an area consisting of a specified pixel and itssurrounding pixels; (c) selecting a correlation judgment method and apixel interpolation method on said specified pixel on the basis of saidchroma evaluation value; and (d) performing a pixel interpolationprocess on said specified pixel in a correlation direction determined bythe selected correlation judgment method, by using the selected pixelinterpolation method, wherein by using predetermined two thresholdvalues TH1 and TH2 (TH1≧TH2), if said chroma evaluation value is notlarger than the threshold value TH2, a correlation judgment method forgray image and a pixel interpolation method for gray image are selected,if said chroma evaluation value is larger than the threshold value TH2and not larger than the threshold value TH1, a correlation judgmentmethod using a correlation value selected out of correlation values forgray image and color image and a pixel interpolation method for colorimage are selected, and if said chroma evaluation value is larger thanthe threshold value TH1, a correlation judgment method for color imageand a pixel interpolation method for color image are selected in saidstep (c).
 2. The pixel interpolation method according to claim 1,wherein said step (c) includes the step of (c-1) selecting a correlationjudgment method using a correlation value out of correlation values forgray image and color image, which is judged that the correlation on saidspecified pixel is reflected higher on, if said chroma evaluation valueis larger than the threshold value TH2 and not larger than the thresholdvalue TH1.
 3. The pixel interpolation method according to claim 2,wherein in a first direction and a second direction which are orthogonalto each other, a differential absolute value A1 between a correlationvalue for gray image in the first direction and a correlation value forgray image in the second direction and a differential absolute value A2between a correlation value for color image in the first direction and acorrelation value for color image in the second direction are calculatedand if a differential absolute value A3 between the differentialabsolute value A1 and the differential absolute value A2 is not largerthan a predetermined threshold value THA, a correlation value out of thecorrelation values for gray image and color image, whose correlation ishigher, is used in the first direction and the second direction in saidstep (c-1).
 4. The pixel interpolation method according to claim 3,wherein said first direction is a vertical direction and said seconddirection is a horizontal direction.
 5. The pixel interpolation methodaccording to claim 3, wherein said first direction is a direction havingthe inclination of 45 degrees with respect to the horizontal direction.6. The pixel interpolation method according to claim 2, wherein in afirst direction and a second direction which are orthogonal to eachother, a differential absolute value A1 between a correlation value forgray image in the first direction and a correlation value for gray imagein the second direction and a differential absolute value A2 between acorrelation value for color image in the first direction and acorrelation value for color image in the second direction are calculatedand if a differential absolute value A3 between the differentialabsolute value A1 and the differential absolute value A2 is larger thana predetermined threshold value THA, a correlation value for gray imageis used in the first direction and the second direction when thedifferential absolute value A1 is larger than the differential absolutevalue A2 and a correlation value for color image is used in the firstdirection and the second direction when the differential absolute valueA2 is larger than the differential absolute value A1 in said step (c-1).7. The pixel interpolation method according to claim 6, wherein saidfirst direction is a vertical direction and said second direction is ahorizontal direction.
 8. The pixel interpolation method according toclaim 6, wherein said first direction is a direction having theinclination of 45 degrees with respect to the horizontal direction. 9.The pixel interpolation method according to claim 1, wherein said step(b) comprises the steps of: (b-1) calculating a first color differencecomponent evaluation value from a color difference component value onthe basis of respective average pixel values for color components byusing said specified pixel and said surrounding pixels; (b-2)calculating a second color difference component evaluation value byaccumulating color difference component values in a predetermineddirection by using said specified pixel and said surrounding pixels; and(b-3) comparing said first color difference component evaluation valueand said second color difference component evaluation value to selectone whose level of color difference component is smaller as said chromaevaluation value.
 10. The pixel interpolation method according to claim9, wherein said step (b-2) comprises the steps of: (b-2-1) calculating avertical color difference component evaluation value by accumulatingcolor difference component values in a vertical direction by using saidspecified pixel and said surrounding pixels; and (b-2-2) calculating ahorizontal color difference component evaluation value by accumulatingcolor difference component values in a horizontal direction by usingsaid specified pixel and said surrounding pixels, said step (b-3)comprises the step of: (b-3-1) comparing said first color differencecomponent evaluation value, said vertical color difference componentevaluation value and said horizontal color difference componentevaluation value to select one whose level of color difference componentis smallest as said chroma evaluation value.
 11. A pixel interpolationmethod comprising the steps of: (a) inputting a pixel signal of apredetermined color space; (b) calculating a chroma evaluation value ofan area consisting of a specified pixel and its surrounding pixels; (c)selecting a correlation judgment method and a pixel interpolation methodon said specified pixel on the basis of said chroma evaluation value;and (d) performing a pixel interpolation process on said specified pixelin a correlation direction determined by the selected correlationjudgment method, by using the selected pixel interpolation method,wherein by using predetermined three threshold values TH1, THM and TH2(TH1≧THM≧TH2), if said chroma evaluation value is not larger than thethreshold value TH2, a correlation judgment method for gray image and apixel interpolation method for gray image are selected, if said chromaevaluation value is larger than the threshold value TH2 and not largerthan the threshold value THM, a correlation judgment method for grayimage and a pixel interpolation method for color image are selected, ifsaid chroma evaluation value is larger than the threshold value THM andnot larger than the threshold value TH1, a correlation judgment methodusing a correlation value selected out of correlation values for grayimage and color image and a pixel interpolation method for color imageare selected, and if said chroma evaluation value is larger than thethreshold value TH1, a correlation judgment method for color image and apixel interpolation method for color image are selected in said step(c).
 12. The pixel interpolation method according to claim 11, whereinsaid step (c) includes the step of (c-1) selecting a correlationjudgment method using a correlation value out of correlation values forgray image and color image, which is judged that the correlation on saidspecified pixel is reflected higher on, if said chroma evaluation valueis larger than the threshold value THM and not larger than the thresholdvalue TH1.
 13. The pixel interpolation method according to claim 12,wherein in a first direction and a second direction which are orthogonalto each other, a differential absolute value A1 between a correlationvalue for gray image in the first direction and a correlation value forgray image in the second direction and a differential absolute value A2between a correlation value for color image in the first direction and acorrelation value for color image in the second direction are calculatedand if a differential absolute value A3 between the differentialabsolute value A1 and the differential absolute value A2 is not largerthan a predetermined threshold value THA, a correlation value out of thecorrelation values for gray image and color image, whose correlation ishigher, is used in the first direction and the second direction in saidstep (c-1).
 14. The pixel interpolation method according to claim 13,wherein said first direction is a vertical direction and said seconddirection is a horizontal direction.
 15. The pixel interpolation methodaccording to claim 13, wherein said first direction is a directionhaving the inclination of 45 degrees with respect to the horizontaldirection.
 16. The pixel interpolation method according to claim 12,wherein in a first direction and a second direction which are orthogonalto each other, a differential absolute value A1 between a correlationvalue for gray image in the first direction and a correlation value forgray image in the second direction and a differential absolute value A2between a correlation value for color image in the first direction and acorrelation value for color image in the second direction are calculatedand if a differential absolute value A3 between the differentialabsolute value A1 and the differential absolute value A2 is larger thana predetermined threshold value THA, a correlation value for gray imageis used in the first direction and the second direction when thedifferential absolute value A1 is larger than the differential absolutevalue A2 and a correlation value for color image is used in the firstdirection and the second direction when the differential absolute valueA2 is larger than the differential absolute value A1 in said step (c-1).17. The pixel interpolation method according to claim 16, wherein saidfirst direction is a vertical direction and said second direction is ahorizontal direction.
 18. The pixel interpolation method according toclaim 16, wherein said first direction is a direction having theinclination of 45 degrees with respect to the horizontal direction. 19.The pixel interpolation method according to claim 11, wherein said step(b) comprises the steps of: (b-1) calculating a first color differencecomponent evaluation value from a color difference component value onthe basis of respective average pixel values for color components byusing said specified pixel and said surrounding pixels; (b-2)calculating a second color difference component evaluation value byaccumulating color difference component values in a predetermineddirection by using said specified pixel and said surrounding pixels; and(b-3) comparing said first color difference component evaluation valueand said second color difference component evaluation value to selectone whose level of color difference component is smaller as said chromaevaluation value.
 20. The pixel interpolation method according to claim19, wherein said step (b-2) comprises the steps of: (b-2-1) calculatinga vertical color difference component evaluation value by accumulatingcolor difference component values in a vertical direction by using saidspecified pixel and said surrounding pixels; and (b-2-2) calculating ahorizontal color difference component evaluation value by accumulatingcolor difference component values in a horizontal direction by usingsaid specified pixel and said surrounding pixels, said step (b-3)comprises the step of: (b-3-1) comparing said first color differencecomponent evaluation value, said vertical color difference componentevaluation value and said horizontal color difference componentevaluation value to select one whose level of color difference componentis smallest as said chroma evaluation value.